Amphibious robotic device

ABSTRACT

A robotic device for navigating in at least a liquid medium, includes a legged propulsion system having a series of legs external of a body of the robotic device, each of the legs being independently driven and mounted to the body for pivotal movement about a respective transverse axis. The legs oscillating relative to the body about the respective transverse axis such that interaction between the legs and the liquid medium produces propulsive forces that displace the robotic device within the liquid medium. A control system is operatively connected to the legged propulsion system for autonomous control and operation of the robotic device based on information received from at least one sensor providing data about an environment of the device.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 11/497,302 filed Aug. 2, 2006, the entire contents of which isincorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to robotic devices, particularly torobotic devices designed to maneuver in a liquid medium as well as on asolid medium.

BACKGROUND ART

In general, underwater robotics poses certain unique challenges thatrender many of the principles of terrestrial robotics problematic. Arobot underwater is able to move along six degrees of freedom, andmaneuvering with six degrees of freedom creates serious complications. Acomputationally straightforward task of pose maintenance on land becomesfar more challenging under water, because of environmental factors suchas strong currents in marine environments. Infra-red sensors lose someof their effectiveness in water as well. Wireless radio communicationsare also impossible over a large distance in water compared to groundbased control. All these issues make underwater robotics problems moredifficult than terrestrial robotics.

The traditional approach used to propel undersea vehicles is by usingpropellers or thrusters. Although simple by design, these vehicles lackthe maneuverability and agility seen in fish and other marine species.In addition, thrusters are not an energy efficient approach to stationkeeping underwater.

In computer vision, visual tracking is the process of repeatedlycomputing a position of a feature or sets of features in a sequence ofinput images. A number of methods for visual tracking in a dryenvironment (i.e. not underwater) based on the color of the target areknown. One of the known approaches is color-blob tracking, where thetrackers segment out sections of the image that match a threshold levelfor the given target and based on the segmentation output, tracks theshape, size or centroid of the blob, among other features. Anotherapproach is the matching of color histograms, which are a measure ofcolor distribution over an image. Some of the tracking methods arecombined with statistical methods to provide more accurate results, oneexample being mean-shift tracking algorithms, which attempt to maximizethe statistical correlation between two distributions. However, thetracking of objects in a dry environment is very different from thetracking of objects underwater. Underwater, vision is impaired by theturbidity of the water caused by floating sedimentation (“aquatic snow”)and other floating debris. The behavior of the light beams is altered bymany factors including refraction, which is influenced by waves andwater salinity level, scattering, which causes a reduction of contrastbetween objects and influences color hues, and absorption, which isfrequency dependent and makes detection of certain colors difficult. Assuch, vision in underwater environments has rarely been examined due tothe complications involved.

SUMMARY OF INVENTION

It is therefore an aim of the present invention to provide an improvedrobotic device.

Therefore, in accordance with the present invention, there is provided arobotic device for navigating in at least a liquid medium, the roboticdevice comprising: a legged propulsion system having a series of legsexternal of a body of the robotic device, each of the legs beingindependently driven and mounted to the body for pivotal movement abouta respective transverse axis, each of the legs being operable to atleast oscillate relative to the body about the respective transverseaxis such that interaction between the legs and the liquid mediumproduces propulsive forces that displace the robotic device within theliquid medium; and a control system operatively connected to the leggedpropulsion system for autonomous control and operation of the roboticdevice based on information received from at least one sensor providingdata about an environment of the device, the control system using datafrom the at least one sensor to determine a desired motion of therobotic device and a corresponding required leg motion of each of thelegs to produce the desired motion, and the control system autonomouslyactuating each of the legs of the legged propulsion system in accordancewith the corresponding required leg motion.

There is also provided, in accordance with the present invention, anamphibious robotic device comprising: a legged propulsion system havinga series of legs, each of said legs being driven by an actuator andmounted for pivotal movement about a respective transverse axis in oneof at least a swimming mode and a walking mode, said legs beingconfigured to pivotally oscillate relative to the transverse axis insaid swimming mode when the device is in a liquid medium such thatinteraction between said legs and the liquid medium provides propulsiveforces that displace the vehicle body within the liquid medium, saidlegs being configured to rotate relative to the transverse axis in saidwalking mode when the device is on a solid medium such that interactionbetween said legs and the solid medium provides propulsive forces thatdisplace the vehicle body in a desired direction on the solid medium;and a control system having at least one sensor operable to autonomouslydetect with which of the liquid medium and the solid medium the roboticdevice is interacting and a leg controller synchronously operating saidlegs in either one of the swimming mode and the walking mode based onthe detected medium.

There is further provided, in accordance with the present invention, acontrol system for autonomously maneuvering a robotic device in at leastone of a liquid medium and a solid medium, the robotic device includinga propulsion system having a series of individually controlled legs, thecontrol system comprising: at least one visual sensor retrieving animage of an environment of the device in the medium; an image analyzingmodule receiving the image, determining a presence of an object of agiven type therein and analyzing at least one property of the object; amotion calculator determining a desired motion of the device based onthe at least one property of the object; and a controller operating thepropulsion system of the device, the controller calculating a respectiverequired leg motion of each of the legs to obtain the desired motion ofthe device and operating each of the legs based on the respectiverequired leg motion calculated, such that the robotic deviceautonomously maneuvers in said medium.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made to the accompanying drawings, showing by wayof illustration a particular embodiment of the present invention and inwhich:

FIG. 1 is a perspective view of a robotic device in accordance with aparticular embodiment of the present invention;

FIG. 2 is a top view of the device of FIG. 1, with a top panel thereofremoved for improved clarity;

FIG. 3 is a perspective view of a leg of the device in accordance withan alternate embodiment of the present invention;

FIG. 4 is a cross-sectional view of the leg of FIG. 3;

FIG. 5 is a perspective view of a leg of the device in accordance withanother alternate embodiment of the present invention;

FIG. 6 is a cross-sectional view of the leg of FIG. 5;

FIG. 7 is a perspective view of a leg of the device in accordance with afurther alternate embodiment of the present invention;

FIG. 8 is a cross-sectional view of the leg of FIG. 7;

FIG. 9 is a block diagram of a control system for the device of FIG. 1;

FIG. 10 is a block diagram of a particular embodiment of the controlsystem of FIG. 9;

FIG. 11 is a block diagram of another particular embodiment of thecontrol system of FIG. 9;

FIG. 12A is a schematic top view of the device of FIG. 1 showingparameters used in a hovering gait thereof;

FIG. 12B is a schematic side view of the device of FIG. 1 illustrating astand-by range and a thrust range of the hovering gait; and

FIG. 12C is a schematic, partial side view of the device of FIG. 1illustrating the angle, thrust and pressure drag for one of the legs.

DETAILED DESCRIPTION OF PARTICULAR EMBODIMENTS

Referring to FIGS. 1-2, a robotic device in accordance with a particularembodiment of the present invention is generally shown at 10. The device10 is designed as an aquatic swimming robot that is also capable ofoperating on a solid medium, including compact ground surfaces and sand.As such, the device 10 is said to be amphibious in that it can walk on asolid surface and penetrate a neighboring liquid medium, swim in thatliquid medium and exit the liquid medium on an appropriate interfacebetween the solid and liquid medium (e.g. a ramped up bottom surface ofthe liquid environment becoming the solid medium such as a beach).

The device 10 comprises a body 12 including a waterproof shell 14 whichcan be for example made of aluminum, inside which all electronics,sensors, power supplies, actuators, etc. are housed. The shell 14includes buoyancy plates which allows for the buoyancy of the device 10to be adjusted depending on the properties of the liquid medium thedevice 10 is going to be submerged in (e.g. salt water, fresh water).The buoyancy of the device 10 is preferably adjusted to near neutral.

The device 10 comprises six legs 16 a which are attached to the bodysuch as to be rotatable about a respective rotational axis 18 extendingtransversely with respect to the body 12. As such, the motion of eachleg 16 a is controlled by a single respective actuator 20 (see FIG. 2),thus minimizing the complexity of the motion of the device 10 as well asthe required energy to actuate the legs 16 a. Underwater, the legs 16 agive the device the ability to turn sideways (yaw), dive (pitch) androtate on its horizontal axis (roll), as well as the ability to moveforward and backward (surge), and up and down (heave).

The device 10 comprises a front camera assembly 90 mounted in the frontof the body 12 and a rear camera assembly 92 mounted in the rear of thebody 12. The device also includes a power source, for example a seriesof batteries 94 (see FIG. 2). The device 10 can further comprise variousinternal sensors for monitoring the device functions, for examplebattery power and power consumption levels for the leg actuators 20.

Leg Design

In the embodiment of FIGS. 1-2, the legs 16 a shown are mainly designedfor water propulsion. Each actuator 20 is mounted rigidly to acylindrical hip extension 22, 22′ which is in turn rigidly mounted tothe body 12. Each leg 16 a includes a leg mount 24 which is rigidlymounted on an output shaft 25 (see FIG. 1) of the respective actuator 20passing through the respective hip extension 22, 22′, such that the legmount 24 is rotated by the output shaft 25. Each leg 16 a also includesa flipper 26 which extends from the leg mount 24 such that alongitudinal axis 27 (see FIG. 2) of the flipper 26 is perpendicular orsubstantially perpendicular to the rotational axis 18 of the respectiveleg 16 a. The legs 16 a produce thrust underwater through an oscillatorymotion, i.e. a reciprocating pivotal motion of each flipper 26 about itsrespective rotational axis 18. In a particular embodiment, each flipper26 includes two flexible sheets, for example made of vinyl, which areglued together over a series of rods. The flexible sheets provide thenecessary flexibility to capture water when the legs 16 a areoscillating, while the rods provide the necessary stiffness to generatethrust. The legs 16 a are distributed symmetrically about the body 12,with three legs 16 a being equally spaced apart on each side thereof.The middle hip extensions 22′ are longer than the front and rear hipextensions 22, such that the middle legs 16 a extend outwardly of theother legs 16 a to avoid interference between adjacent flippers 26. Inan alternate embodiment, the relative size of the body 12 and theflippers 26 is such that the hip extensions 22 have a similar lengthwithout interference between the legs 16 a of a same side. While thelegs 16 a are particularly well suited for use underwater, they are lessefficient for motion on a solid medium.

Referring to FIGS. 3-4, an amphibious leg 16 b according to a particularembodiment of the present invention is shown. This leg 16 b allows for agood compromise between underwater propulsion and propulsion on a solidmedium. By replacing the legs 18 a of the device 10 with legs such asthe leg 18 b, good underwater propulsion is provided, although generallyless efficient than with the legs 16 a, and efficient propulsion on asolid medium is provided, both forward and backward. Each leg 16 bincludes a leg mount 24 similar to that of the previously described legs16 a, and which rigidly engages the respective actuator output shaft 25(not shown in FIGS. 3-4) for rotation and/or oscillation about therespective axis 18. An upper member 30 b of the leg 16 b includes anattachment plate 32 rigidly attached to the leg mount 24, and spacedapart upper rigid plates 34 b extending from the attachment plate 32perpendicularly to the rotational axis 18 of the leg 16 b. The upperplates 34 b are teardrop-shaped, with their distal larger end 38 b beinginterconnected by two spaced apart transverse plates 40 b. A joint pin42 b also extends between the distal ends 38 b of the upper plates 34 bbetween the transverse plates 40 b and the attachment plate 32.

A lower member 44 b of the leg 16 b includes lower spaced apart rigidplates 46 b also extending perpendicularly to the rotational axis 18 ofthe leg 16 b. The proximal ends 48 b of the lower plates 46 b extendoutwardly of the distal ends 38 b of the upper plates 34 b and are alsointerconnected by the joint pin 42 b. The joint pin 42 b is rotationallyconnected to at least one of the upper and lower members 30 b, 44 b,such that the members 30 b, 44 b are pivotally interconnected by thejoint pin 42 b. The lower plates 46 b are also teardrop-shaped, withtheir distal larger end 50 b being interconnected by two spaced apartend pins 52 b, and by a rounded end connecting member 54.

Each of the upper plates 34 b forms with its associated lower plate 46 ba substantially planar frame 56 b mainly defined in a planeperpendicular to the rotational axis 18 of the leg 16 b, and as suchcreating minimal interference with the water when the device 10 issubmerged. The frames 56 b are made of a rigid material which can be forexample an adequate metal or composite material.

The leg 16 b also includes an elastic member 58 b which extends betweenthe frames 56 b perpendicularly thereto. Referring particularly to FIG.4, the elastic member 58 b is a double member which extends from theattachment plate 32 to around the end pins 52 b, passing between thetransverse plates 40 b and on each side of the joint pin 42 b. Theelastic member 58 b is a thrust producing member when the leg 16 b ismoved underwater. The spaced apart end pins 52 b shape the elasticmember 58 b so that during swimming, the liquid medium is forced awayfrom the end of the lower section 54, thus reducing the drag. Thetransverse plates 40 b offset the bending point of the elastic member 58b from the joint pin 42 b, such as to increase the amount by which theelastic member 58 b stretches upon relative pivoting of the members 30b, 44 b. The elastic member 58 b thus provides compliance to the leg 16b, while limiting the relative pivoting motion of the upper and lowermembers 30 b, 44 b such that the frames 56 b can bear the weight of thedevice 10 when the device 10 moves on solid ground. In a particularembodiment, the elastic member 58 b is made of a material providingincreased resistance to the pivoting motion of the members 30 b, 44 babout the joint pin 42 b as the members 30 b, 44 b are pivoted away fromthe aligned position shown in the Figures.

The leg 16 b therefore defines a jointed limb that is at least partiallycompliant when used on a solid medium and at least partially stiff whenused in a liquid medium. The elastic member 58 b acts similarly to aligament which interconnects the two members 30 b, 44 b of the leg 16 band constrains the relative pivotal movement such as to arrive at adesired gait of the device, both on a solid medium and in a liquidmedium (e.g. water).

Referring to FIGS. 5-6, an amphibious leg 16 c according to an alternateembodiment of the present invention is shown. Each leg 16 c includes aleg mount 24 (not shown in FIGS. 5-6) is similar to that of thepreviously described legs 16 a,b and which rigidly engages therespective actuator output shaft 25 (not shown in FIGS. 5-6). Anattachment plate 32 is rigidly attached to the leg mount 24, and spacedapart rigid plates 34 c extend from the attachment plate 32perpendicularly to the rotational axis 18 of the leg 16 c. The plates 34c have rounded distal ends 38 c interconnected at each extremity thereofby a transverse plate 40 c. A flexible toe plate 60 is rigidly connectedto each transverse plate 40 c, the toe plate 60 flexing upon contact ofthe leg 16 c with a solid surface to augment a contact area between theleg 16 c and the solid surface, and springing back into an unflexedposition when contact with the ground is lost. The plates 40 c thusdefine spaced apart rigid frames 56 c which, by being mainly defined ina plane perpendicular to the rotational axis 18 of the leg 16 c, have aminimal interference with the water when the device 10 is submerged.

The leg 16 c also includes a flexible flipper 62, the proximal end 64thereof being rigidly connected to the attachment plate 32 and thedistal end 66 thereof being free. The flipper 62 extends between therigid frames 56 c perpendicularly thereto. Because of its flexibility,the flipper 62 is free to flap between the rigid frames 56 c. Theflipper 62 is shorter than the rigid frames 56 c such as to avoidcontact with the ground in a dry environment. The flipper 62 and toeplates 60 are thrust producing members when the leg 16 c is movedunderwater. The rigid frames 56 c provide a channeling effect underwatersuch that there is not spill over when the flipper 62 bends, thusincreasing the thrust produced.

Referring to FIGS. 7-8, an amphibious leg 16 d according to a furtheralternate embodiment of the present invention is shown. This leg 16 dhas however limited efficiency in a backward movement on land because ofits directionally limited pivoting motion, as will be described furtherbelow. Each leg 16 d includes a leg mount 24 (not shown in FIGS. 7-8)which is similar to that of the previously described legs 16 a,b,c, andwhich rigidly engages the respective actuator output shaft 25 (not shownin FIGS. 7-8). An upper member 30 d includes an attachment plate 32rigidly attached to the leg mount 24, and spaced apart upper rigidplates 34 d extending from the attachment plate 32 perpendicularly tothe rotational axis 18 of the leg 16 d. In the embodiment shown, theupper plates are 34 d oblong shaped, although any shape conducive togood fluid flow underwater while maintaining adequate strength andrigidity when loaded by the weight of the device 10 on land can be used.The proximal end 36 d of the upper plates 34 d is interconnected byfirst and second spaced apart end pins 68, 69. An upper joint pin 42 dalso extends between the distal ends 38 d of the upper plates 34 d.

A middle member 70 of the leg 16 d includes middle spaced apart rigidplates 72 also extending perpendicularly to the rotational axis 18 ofthe leg 16 d. In the embodiment shown the middle plates 72 are alsooblong shaped, although here again any shape conductive to good fluidflow underwater while maintaining adequate strength and rigidity whenloaded by the weight of the device 10 on land can be used. The proximalends 74 of the middle plates 72 extend inwardly of the upper plates 34 dand are also interconnected by the upper joint pin 42 d. The upper jointpin 42 d is rotationally connected to at least one of the upper andmiddle members 30 d, 70, such that the upper and middle members 30 d, 70are pivotally interconnected by the upper joint pin 42 d. The proximalends 74 of the middle plates 72 also include a stopper 76 which limitsthe pivoting motion between the upper and middle members 30 d, 70 alonga single direction from the aligned position shown in the Figures. Thedistal ends 78 of the middle plates 72 are interconnected by a lowerjoint pin 80.

A lower member 44 d of the leg 16 d includes lower spaced apart rigidplates 46 d also extending perpendicularly to the rotational axis 18 ofthe leg 16 d. The lower plates 46 d are semi-circular, with a roundededge 82 defining a contact area with the ground surface. The lowerplates 46 d, along a proximal end 48 d of the rounded edge 82, extendoutwardly of the middle plates 72 and are also interconnected by thelower joint pin 80. The lower joint pin 80 is rotationally connected toat least one of the middle and lower members 70, 44 d, such that themiddle and lower members 70, 44 d are pivotally interconnected by thelower joint pin 80. The distal ends 78 of the middle plates 72 include astopper 81 which limits the pivoting motion between the middle and lowermembers 70, 44 d along a single direction from the aligned positionshown in the Figures. The lower plates 46 d, along a distal end 50 d ofthe rounded edge 82, are interconnected by a third end pin 52 d.

Each set of connected upper, middle and lower members 30 d, 70, 44 dthus defines a substantially planar frame 56 d, the two apart frames 56d being mainly defined in a plane perpendicular to the rotational axis18 of the leg 16 d and as such having a minimal interference with thewater when the device is submerged.

The leg 16 d also includes an elastic member 58 d which extends betweenthe frames 56 d perpendicularly thereto. The elastic member 58 d is adouble member which extends around the first end pin 68 and the thirdend pin 52 d, passing in between in a zigzag pattern against the secondend pin 69, against the upper joint pin 42 d on a side thereof oppositethat of the second end pin 69 and against the lower joint pin 80 on sidethereof opposite that of the upper joint pin 42 d. The elastic member 58d is a thrust producing member when the leg 16 d is moved underwater.The elastic member 58 d also provides compliance to the leg 16 d whilelimiting the relative pivoting motion between the members 30 d, 70, 44 dabout the joint pins 42 d, 80, by forcing the leg 16 d in the alignedposition, shown in the Figures, against the stoppers 76, 81. As such,the frames 56 d can bear the weight of the device 10 when the device 10moves on solid ground. In a particular embodiment, the elastic member 58d is made of a material providing increased resistance to the pivotingmotion of the members 30 d, 70, 44 d about the joint pins 42 d, 80 asthe members 30 d, 70, 44 d are pivoted away from the aligned position.

Control System

Referring to FIG. 9, a control system 102 for the device 10 is shown.The system 102 optionally includes an operator control unit 104 allowingan operator to directly control the device 10, the operator control unit104 sending signals to a motion calculator 106 in accordance with theoperator input. The operator control unit 104 receives feedback from atleast one visual sensor 108 of a visual control system 120, which in aparticular embodiment includes a camera from one or both of the frontand rear camera assemblies 90, 92, which provides streaming video fromthe device 10. The operator control unit 104 also receives feedback froman inertial sensor 110, or Inertial Measurement Unit (IMU), installedwithin the body 12 for orientation and acceleration sensing. Theoperator control unit 104 can also receive feedback from the motioncalculator 106 containing any other relevant data to assist incontrolling the device 10. Communication between the device 10 and theoperator control unit 104 is optionally done over a fiber optic tether(not shown).

As described further below, the visual control system 120 also controlsthe device 10, and as such the operator control unit 104 can be omittedor used in conjunction with the visual control system 120.

The motion calculator 106 computes a desired motion of the device 10,i.e. pitch, roll, yaw, heave and/or surge, based for example on thesignal from the operator control unit 104, and communicates this desiredmotion to a leg controller 112.

The leg controller 112 computes a required thrust at each leg 16 a,b,c,dto obtain the desired, motion, and determines the corresponding motionfor each leg 16 a,b,c,d. The leg controller 112 moves the legs 16a,b,c,d in accordance with a series of preset gaits programmed therein.Gaits are a combination of leg parameters that generate a fixed motionfor a fixed set of parameters. Depending on whether the device 10 isswimming or walking, different table-driven gaits are used to drive thedevice 10. Walking gaits move the device 10 through complete rotation ofthe legs 16 a,b,c,d. Swimming gaits move the device through a “kicking”motion of the legs 16 a,b,c,d, i.e. an oscillatory motion of the legs 16a,b,c,d with various phase and amplitude offsets. In both cases, the legcontroller 112 computes a series of three parameters for each leg 16a,b,c,d based on the required leg thrust: amplitude, offset and phase.The leg controller 112 actuates the actuator 20 of each leg 16 a,b,c,dbased on the three parameters calculated. The amplitude parametergoverns the distance the legs 16 a,b,c,d sweep along the spherical archdefined around the rotational axis 18 during each cycle. Offset dictatesthe starting orientation of the legs 16 a,b,c,d relative to each otherat the beginning of the cycle. Direction of the leg motion is controlledby the phase parameters of each leg 16 a,b,c,d.

However, the swimming gaits only allow the device 10 to turn if it ismoving forward or backward. As such, the leg controller 112 alsoincludes hovering gaits to allow the device 10 to move along 5 degreesof freedom, i.e. pitch, roll, yaw, heave and/or surge without forward orrearward movement, such as to be able to hold a fixed position despitewater currents, and to turn at that fixed location, for example to keepan object within camera range.

The motion calculator 106 also receives inertial data from the inertialsensor 110 which can influence the required motion communicated to theleg controller 112. In a particular embodiment, the motion calculator106 uses the inertial data to determine if the device 10 is in a dry orunderwater environment, and directs the leg controller 112 to use eitherthe walking or the swimming/hovering gaits accordingly. As such thedevice 10 can autonomously switch to the appropriate gaits upon enteringor coming out of the water.

Hovering Gaits

The hovering gaits of the leg controller 112, which allow the device 10to perform station keeping, are described below. The leg controller 112receives the required motion input from the motion calculator 106including pitch (Cp), roll (Cr), yaw (Cy), heave (Ch) and surge (Cs).The leg controller first computes Fx and Fz, the column vectorsrepresenting the desired thrust at each leg location in the x and zdirections (see FIG. 12A, where arrow 114 indicates the forwarddirection). The legs 16 a,b,c,d cannot generate thrust in the ydirection. Accordingly, the desired thrust Fx and Fz for each leg 16a,b,c,d is computed by the leg controller 112 according to thefollowing:

$F_{x} = {\begin{pmatrix}0 & 0 & {- k_{y}} & 0 & k_{s} \\0 & 0 & {- k_{y}} & 0 & k_{s} \\0 & 0 & {- k_{y}} & 0 & k_{s} \\0 & 0 & k_{y} & 0 & k_{s} \\0 & 0 & k_{y} & 0 & k_{s} \\0 & 0 & k_{y} & 0 & k_{s}\end{pmatrix} \times C}$ $F_{z} = {\begin{pmatrix}k_{p} & k_{r} & 0 & k_{h} & 0 \\0 & k_{r} & 0 & k_{h} & 0 \\{- k_{p}} & k_{r} & 0 & k_{h} & 0 \\{- k_{p}} & {- k_{r}} & 0 & k_{h} & 0 \\0 & {- k_{r}} & 0 & k_{h} & 0 \\k_{p} & {- k_{r}} & 0 & k_{h} & 0\end{pmatrix} \times C}$

where C=[C_(p) C_(r) C_(y) C_(h) C_(s)]^(T) and the constants kp, kr,ky, kh and ks are used to scale down the input so that the absolutemaximum value of the output is less than or equal to 1. The size of thecolumn vectors Fx and Fz is equal to the number of legs 16 a,b,c,d onthe device 10, which in this case is six (6). The legs 16 a,b,c,d areshown in FIG. 12A as identified by a number n ranging from 0 to 5, thelegs 0 and 5 facing forward to provide an extended moment arm andsymmetric pitching moment with the legs 2 and 3, as well as to providequick reverse surge.

The selected thrust angle θc,n and magnitude Tc,n for each leg n is thuscomputed by the leg controller 112 as:

$\theta_{c,n} = {{{\arctan \left( \frac{F_{zn}}{F_{xn}} \right)}\mspace{14mu} {and}\mspace{14mu} T_{c,n}} = \sqrt{F_{xn}^{2} + F_{zn}^{2}}}$

The most efficient way to generate thrust in a direction θr is throughrapid oscillation (period of 250-500 ms) of the leg 16 a,b,c,d aroundthat angle θr. The magnitude of the thrust is approximately proportionalto the amplitude of the oscillation. The leg angle θf over time is thus:

θ_(f)(t)=T _(n) sin(ω_(f) t+φ _(f))+θ_(r)

where ωf is the fixed frequency of oscillation and a phase shift φ_(f)is used between the legs. The leg angle θf and thrust Tn are shown inFIG. 12C for one of the legs 16 a,b,c,d.

To minimize delay in the execution of commands and parasitic forcesgenerated upon orientation of the legs 16 a,b,c,d, the leg controller112 limits the range of thrust angle for each leg to a region 116 (seeFIG. 12B) of 180°. This reduces the average reorientation angleresponsible for the parasitic forces at the cost of reduced maximumdevice thrust. For example, the front legs are not used when a forwardthrust is commanded, thereby reducing the maximum possible forwardthrust of the device 10. To further improve the reaction time of thedevice 10, the leg controller 112 uses the pressure drag forces D (seeFIG. 12C) generated when the legs 16 a,b,c,d are reoriented. When thedifference between the desired thrust angle and the current leg angle isgreater than 45°, the leg 16 a,b,c,d is rotated at a rate that generatesa pressure drag D consistent with the desired thrust Tc via a constantKPD. As the leg surface passes the 45° region, the oscillation amplitudeis increased until it reaches its selected amplitude as given by theequation for θf(t) set out above. Using discrete time equations andletting θr be the ramped value of the computed thrust angle θc andmagnitude Tc, the motion for each leg 16 a,b,c,d is set by the legcontroller 112 in accordance with the two following equations:

$\begin{matrix}{{{\theta \; {r\left\lbrack {t + 1} \right\rbrack}} = {\theta \; {r\lbrack t\rbrack}}}\mspace{14mu}} & {{if}\mspace{14mu} \theta \; {c\lbrack t\rbrack}\mspace{11mu} {outside}\mspace{14mu} {of}\mspace{14mu} {thrust}\mspace{14mu} {range}} \\{{ramp}\left( {{KPDTc},{\theta \; {r\lbrack t\rbrack}},{\theta \; {c\lbrack t\rbrack}}} \right)} & {otherwise} \\{{\theta \; {f\left\lbrack {t + 1} \right\rbrack}} = {\theta \; {r\lbrack r\rbrack}}} & {{{if}{{{\theta_{c}\lbrack t\rbrack} - {\theta_{f}\lbrack t\rbrack}}}} > {45{^\circ}}} \\{{T_{c}\frac{{{\theta_{r}\lbrack t\rbrack} - {\theta_{f}\lbrack t\rbrack}}}{45{^\circ}}{\sin \left( {{\omega_{f}t} + \varphi_{f}} \right)}} + {\theta_{r}\lbrack t\rbrack}} & {otherwise}\end{matrix}$

where the ramp(rate,a,b) function ramps value b toward a at a constantrate.

Also, in order to improve slow-changing commands, when the demandedthrust Tc reaches zero, the leg controller 112 gradually moves the legs16 a,b,c,d back to a stand-by range 118 (see FIG. 12B) of 90°, such thatthe legs 16 a,b,c,d are always able to generate the proper thrustrapidly by making the leg surface or its normal no more than 45° awayfrom any desired thrust within the thrust range.

The hovering gaits thus allow the device 10 to maintain position basedon operator commands through the operator control unit 104, the visualcontrol system 120 (described further below) and/or input from theinertial sensor 110.

Visual Control Systems

Referring back to FIG. 9, the device 10 also includes the visual controlsystem 120, 120 a, 120 b allowing the device 10 to be controlled basedon visual input, i.e. autonomously from input from the operator controlunit 104. The visual control system 120, 120 a, 120 b includes themotion calculator 106, leg controller 112 and visual sensor 108described above, as well as an image analyzing module 122 receiving datafrom the visual sensor 108. In a particular embodiment, the visualsensor 108 includes a digital camera which is part of the front cameraassembly 90 and which interfaces with the image analyzing module 122.The image analyzing module 122 detects the presence of a target of aselected type, and sends data on at least one property of that target tothe motion calculator 106, as will be described in more detail below. Asdescribed above, the motion calculator 106 computes the required devicemotion and communicated it to the leg controller 112 which controls theleg actuators 20 in accordance with the appropriate gaits.

The target to be recognized by the visual control system 120, 120 a, 120b can be designed to be highly robust and easy to read by both personand the device 10, and can be for example arranged in the form on abooklet and/or on the faces of a geometric object (such as for example acube) to be easy to manipulate and transport.

Referring back to FIG. 2, in a particular embodiment, the device 10includes two computers 124, one for the leg controller 112, and theother for the motion calculator 106 and image analyzing module 122. Bothcomputers 124 are of the PC104/Plus form factor, due to the spacerestrictions inside the body 12. These two computers 124, along withadditional port and interface circuit boards stacked on top of eachother, connect via ISA and PCI buses.

The visual control system 120, 120 a, 120 b can be used with a roboticdevice having a propulsion system other than legs, such as for examplethrusters, the leg controller 112 being replaced by an equivalentcontroller determining the required thrust of the propulsion system andactuating it accordingly.

Tracker-Based Visual Control System

FIG. 10 illustrates a particular embodiment of the visual control system120 a. The image analyzing module 122 includes a visual tracking module126, which receives an image from the visual sensor 108, determines theposition of a given target and calculates an error in the target'sposition through comparison with a desired position signal from adesired position module 128, the desired position usually correspondingto the center of the image. The visual tracking module 126 preferablyuses at least one of a color blob tracker algorithm, a histogram trackeralgorithm and a mean-shift tracker algorithm, which are described below.

The color blob tracker is initialized with the target's color propertiesin the normalized RGB space, which is in effect an over-represented huespace, where the effect of lighting changes common underwater areminimum. This makes the tracking more effective in the underwaterenvironment. The tracker scans the image converted in normalized RGBformat, pixel-by-pixel, and the pixels falling within a given thresholdof the color values of the target are turned on in the output imagewhile the other pixels are turned off. To remove high-frequency (orshot/salt-and-pepper) noise, the median filtering algorithm is used overthe segmented image with either 5-by-5 or 7-by-7 pixel grids, withtypical threshold values of 30%-40%. The tracking algorithm detects theblob in the binary image in every frame, and calculates its centroid,which is taken as the new target location. The total mass of the blob isused as a confidence factor. The error signal is computed using theEuclidean distance in the image between the centroid of the blob and thecenter of the image frame. Two error signals are used for pitch and yaw,and both these signals are sent to the motion controller. A yellowtarget was found to work well with this type of tracker.

The histogram tracker compares rectangular regions of the input framewith the target region by comparing their corresponding colorhistograms. A histogram of the target to be tracked is created andstored. Every image from the camera is divided into rectangular regionsand their normalized histograms having a fixed number of bins over thehue space are calculated. Computationally, the color histogram is formedby discretizing the colors within an image and counting the number ofpixels of each color. Depending on the target and the size of the imageframe, different number of bins can be used, with preferred numbersbeing 32 or 64, and with the regions having either one-eighth orone-sixteenth the dimension of the image frame. The use of normalizedhistograms reduces the effect on color matching of brightness changes inthe underwater environment. The histograms are one-dimensional vectorsthat combine the multi-hue channel data. Similarity between histogramsare computed by known measures, for example the histogram intersectionmeasure, the χ2 (Chi-squared) measure, the Bhattacharyya distancemeasure, or Jeffrey's Divergence. Since the histograms are normalized,the measures return values ranging from 0 to 1; higher values indicatinghigher degree of similarity. The minimum similarity measure ispreferably taken as 0.5; any measure below this threshold is notaccepted as a valid target region. The center of the chosen window istaken as the new target location. As in the case of the color blobtracker, two error signals are used for pitch and yaw, and both thesesignals are sent to the motion controller. This type of tracker issuitable for tracking objects that have a variety of color.

For the mean-shift tracker, color histograms are also used as theunderlying distribution. The histograms are three-dimensional arrays inthis case, one each for the three RGB channels, with preferably 16 binsper channel. The target histogram are computed in a square window ofsides preferably equaling 100 pixels. The color model probabilitydensity function for the target is calculated by overlaying the subwindow by a kernel having the Epanechnikov profile. The weights for themean-shift vector are calculated using the Epanechnikov kernel. Thetracker is initialized with the last known location of the target andthe target PDF model. In each successive tracking step, the candidatewindow having the same size as the target is created at the location ofthe last known target position, the candidate PDF model is calculatedand the weights for pixel are calculated, leading to a new candidateposition. The mean-shift process preferably uses 10 iteration steps tochoose a new target location. The Bhattacharyya distance between thecandidate PDF model and the target PDF model is calculated to quantifythe similarity between the target and the new candidate location. Thelocation with the minimum Bhattacharyya distance is chosen as the newtarget location. As for the other trackers, two error signals (pitch andyaw) are sent to the motion controller. The mean shift tracker isresistant to changes in lighting and appearance of duplicate objects inthe frame, but necessitates substantially more computation than thepreceding trackers.

In all cases, the visual tracking module 126 is able to track the targetat almost 15 frames/second, and therefore without filtering, thecommands sent to the leg actuators 20 by the leg controller 112 would bechanging at such a high rate that it would yield a highly unstableswimming behavior. As such the motion calculator 106 which receives theerror signals from the visual tracking module 126 includes a pitchcontroller 130 and a yaw controller 132 which are both PID controllers,used to take these target locations and produce pitch and yaw commandsat a rate to ensure stable behavior of the device 10. The roll axis isnot affected by this particular type of visual control. Given the inputfrom the visual tracking module 126 at any instant, and the previoustracker inputs, each of the pitch and yaw controllers 130, 132 generatescommands based on the following control law:

$\Delta = {K_{P\; \overset{\_}{ɛ_{i}}} + {K_{I}{\int{\overset{\_}{ɛ_{i}}{t}}}} + {K_{D}\frac{\partial}{\partial t}\overset{\_}{ɛ_{t}}}}$

where ε_(t) is the time-averaged error signal at time t and is definedrecursively as:

ε_(t) =ε_(t)+γ ε_(t-1)

εt is the error signal at time t, KP, KI and KD are respectively theproportional, integral and differential gains and γ is the errorpropagation constant.

The pitch and yaw controllers 130, 132 work identically as follows. Eachcontroller 130, 132 includes a low-pass first-order infinite-impulseresponse or IIR filter (i.e. a digital filter blocking high frequencysignals), smoothing out fast changing pitch and yaw commands byaveraging them over a period of time. A time constant is defined for thelow-pass filter for each controller 130, 132. The gains KP, KI and KDfor each controller 130, 132 are input manually, with limits to truncatethe gains. Each controller 130, 132 has a dead band limit applied to theerror signal, i.e. a range of change in output for which the controller130, 132 will not respond at all. This prevents the controller outputfrom changing too frequently, by ignoring small changes in the errorsignal. A sleep time between each iteration in servoing is alsointroduced to reduce command overhead of the controllers 130, 132.

In a particular embodiment, the parameters for the controllers 130, 132are as follows: a KP of 1.0 with corresponding limit of 1.0, a KI of 0.0with corresponding limit of 0.3, and a KD of 0.0 with correspondinglimit of 1.0 for both controllers 130, 132, a dead band of 0.2 for bothcontrollers 130, 132, a time constant of 0.35 for the pitch controller130 and 0.05 for the yaw controller 132, and a command limit of 1.0 forboth controllers 130, 132.

The pitch and yaw controllers 130, 132 thus send required pitch and yawof the device 10 to the leg controller 112, which as mentioned abovecomputes a required thrust at each leg 16 a,b,c,d, determines acorresponding leg motion following the appropriate gaits, and controlsthe actuator 20 of each leg 16 a,b,c,d accordingly to obtain therequired pitch and yaw.

In an alternate embodiment not shown, the visual tracking module 126also compares the size of the target with a reference size, and sends asize error signal to the motion calculator 106, which computes a desiredspeed change for the device 10, sending corresponding motion data to theleg controller 112. As such the device 10 can remain within a givendistance of the target by modifying its speed.

The visual control system 120 a thus allows the device to follow amoving target or, through use of the hovering gait, hold positionrelative to a stationary target.

Marker-Based Visual Control System

FIG. 11 illustrates another embodiment of the visual control system 120b. The image analyzing module 122 includes a marker detection module134, which is configured to detect a target of a particular type, forexample an ARTag marker. ARTag markers include both symbolic andgeometric content, and are constructed using an error-correcting code toenhance robustness. The marker detection module 134 sends thedescription of the marker, which in the case of an ARTag marker is abinary number corresponding to the black and white regions of themarker, to a marker identification module 136 which is part of themotion calculator 106. The motion calculator 106 also includes a markerlibrary 138, which contains a list of the possible markers, each beingassociated with a particular command, for example turn right, changespeed to 1 m/s, go to location X, film during a given period, switch tothe hovering, swimming or walking gaits, etc. The marker identificationmodule 136 thus accesses the marker library 138 to identify the markerand retrieve the associated command. If the command is one of motion ofthe device 10, the marker identification module 136 sends the requiredmotion signal (pitch, yaw, roll, heave and/or surge) to the legcontroller 112, which as mentioned above computes a required thrust ateach leg 16 a,b,c,d, determines the motion of each leg 16 a,b,c,d basedon the appropriate gaits and controls the actuator 20 of each leg 16a,b,c,d accordingly to produce the required motion. In this embodiment adiver can thus communicate directly with the device 10 and give it aseries of instructions simply by showing it different cards having theadequate markers illustrated thereon. The motion calculator 106 can alsomemorize a series of commands such that the diver can in fact program inadvance a series of motions or tasks for the device 10.

The visual control systems 120 a,b of FIG. 10 and FIG. 11 can be usedtogether, such that for example the device 10 follows a given targetunless a marker is detected, at which point the device 10 stopsfollowing the target and obeys the commands dictated by the marker. Asmentioned above, the visual control system 120, 120 a, 120 b can be usedwith a robotic device moved by a propulsion system other than legs, theleg controller 112 being replaced by an equivalent controller receivingthe desired device motion signal, determining the corresponding requiredthrust of the propulsion system and actuating the propulsion systemaccordingly.

The use of visual sensing to control the device 10 makes use of apassive sensing medium which is thus both non-intrusive as well asenergy efficient. These are both important considerations (in contrastto sonar for example) in a range of applications ranging fromenvironmental assays to security surveillance. Alternative sensing mediasuch as sonar also suffer from several deficiencies which make themdifficult to use for tracking moving targets at close range inpotentially turbulent water.

The visual control abilities of the device 10 allows it to follow amoving object, for instance a diver, and/or accept commands from thediver on presentation of cards carrying predetermined markerscorresponding with tasks to be performed. A complete sequence of actionscan be programmed into the device 10 using the predetermined markers. Assuch a diver can communicate directly with the device 10 without theassistance of an operator located on the surface and as such with orwithout a tether.

The device 10 can thus be operated in a semi-autonomous manner, with orwithout input from an operator on the surface through the operatorcontrol unit 104.

The device 10 of the present invention can be used in a wide range ofapplications. These include underwater search and rescue, coral healthmonitoring, monitoring of underwater establishments (e.g. oil pipelines,communication cables) and many more. Specifically, environmentalassessment tasks in which visual measurements of a marine ecosystem mustbe taken on a regular basis can be performed by the device 10.

The device 10 can also be used in a variety of diver-assisting tasks,such as monitoring divers from a surface, providing lighting (forexample while following a diver), providing communication between diversand the surface, carrying cargo and/or tools, carrying audio equipmentor air reserves, etc.

In a particular embodiment, the device 10 includes an acoustictransducer and as such allows the diver to hear sounds transmitted fromthe surface, stored on the device 10 and/or synthesized by the device10, as well as to send acoustic signals back to the surface by havingthem relayed by the device 10, while following the diver or anothertarget and/or responding to commands given by the diver through the useof visual markers. The sounds could be, for example, music,instructional narrative and/or cautionary information. The soundsemitted by the device 10 can depend on various factors that can besensed by the device 10, for example, on the depth or location of thedevice 10, the length of time the diver has been underwater, the watertemperature, or other environmental parameters.

The visual tracking module 126 can be used to recognize given landmarksand as such allow the device 10 to return autonomously to its startingpoint once a given task is performed. The amphibious legs 16 b,c,d allowthe device to start from and return to a location on dry land whileperforming a task (such as video surveillance) underwater.

The embodiments of the invention described above are intended to beexemplary. Those skilled in the art will therefore appreciate that theforegoing description is illustrative only, and that various alternateconfigurations and modifications can be devised without departing fromthe spirit of the present invention. Accordingly, the present inventionis intended to embrace all such alternate configurations, modificationsand variances which fall within the scope of the appended claims.

1. A robotic device for navigating in at least a liquid medium, therobotic device comprising: a legged propulsion system having a series oflegs external of a body of the robotic device, each of the legs beingindependently driven and mounted to the body for pivotal movement abouta respective transverse axis, each of the legs being operable to atleast oscillate relative to the body about the respective transverseaxis such that interaction between the legs and the liquid mediumproduces propulsive forces that displace the robotic device within theliquid medium; and a control system operatively connected to the leggedpropulsion system for autonomous control and operation of the roboticdevice based on information received from at least one sensor providingdata about an environment of the device, the control system using datafrom the at least one sensor to determine a desired motion of therobotic device and a corresponding required leg motion of each of thelegs to produce the desired motion, and the control system autonomouslyactuating each of the legs of the legged propulsion system in accordancewith the corresponding required leg motion.
 2. The robotic deviceaccording to claim 1, wherein the desired motion includes a series of atleast two consecutive steps, each step including one of movement in atleast one of six degrees of freedom and station keeping.
 3. The roboticdevice according to claim 1, wherein the at least one sensor includes avisual sensor retrieving an image of an environment of the device, thecontrol system determining a presence of an object of a given type inthe image, determining an identity of the object from a given list ofpossible objects of the given type, and determining the desired motionassociated with the identity of the object in the list.
 4. The roboticdevice according to claim 1, wherein the at least one sensor includes avisual sensor retrieving an image of an environment of the device, thecontrol system determining a presence of an object of a given type inthe image, determining a position of the object on the image andcomparing the position to a desired position, and determining thedesired motion of the device such as to change to position to correspondto the desired position.
 5. The robotic device according to claim 1,wherein the control system includes a motion calculator having at leastone angular controller which calculates a required angular displacementnecessary to achieve said desired motion.
 6. The robotic deviceaccording to claim 5, wherein the angular control includes a yawcontroller calculating a required yaw of the device, a pitch controllercalculating a required pitch of the device and a roll controllercalculating a required roll of the device, the desired motion includingat least one of the required yaw, the required pitch and the requiredroll.
 7. The robotic device according to claim 1, wherein each of thelegs is also operable to rotate about the respective transverse axissuch that interaction between the legs and a solid medium allows therobotic device to move on the solid medium, thereby making the roboticdevice amphibious.
 8. The robotic device according to claim 1, whereineach of the legs defines at least two members pivotally interconnectedto relatively pivot about a pivot axis parallel to the respectivetransverse axis.
 9. An amphibious robotic device comprising: a leggedpropulsion system having a series of legs, each of said legs beingdriven by an actuator and mounted for pivotal movement about arespective transverse axis in one of at least a swimming mode and awalking mode, said legs being configured to pivotally oscillate relativeto the transverse axis in said swimming mode when the device is in aliquid medium such that interaction between said legs and the liquidmedium provides propulsive forces that displace the vehicle body withinthe liquid medium, said legs being configured to rotate relative to thetransverse axis in said walking mode when the device is on a solidmedium such that interaction between said legs and the solid mediumprovides propulsive forces that displace the vehicle body in a desireddirection on the solid medium; and a control system having at least onesensor operable to autonomously detect with which of the liquid mediumand the solid medium the robotic device is interacting and a legcontroller synchronously operating said legs in either one of theswimming mode and the walking mode based on the detected medium.
 10. Theamphibious robotic device according to claim 9, wherein each leg definesat least two members pivotally interconnected to relatively pivot abouta pivot axis parallel to the respective transverse axis.
 11. Theamphibious robotic device according to claim 10, wherein each legincludes an elastic material extending through the members and providingresistance to a relative pivoting motion of the members about the pivotaxis.
 12. The amphibious robotic device according to claim 11, whereinthe resistance to the relative pivoting motion of the members increasesas the members pivot away from an aligned position.
 13. The amphibiousrobotic device according to claim 9, wherein the control system isoperatively connected to the legged propulsion system for autonomouscontrol and operation of the robotic device based on informationreceived from the at least one sensor, the control system using datafrom the at least one sensor to determine a desired motion of therobotic device and a corresponding required leg motion of each of thelegs to produce the desired motion, and the control system autonomouslyactuating each of the legs of the legged propulsion system in accordancewith the corresponding required leg motion.
 14. The amphibious roboticdevice according to claim 13, wherein the desired motion includes aseries of at least two consecutive steps, each step including one ofmovement in at least one of six degrees of freedom and station keeping.15. The amphibious robotic device according to claim 13, wherein thecontrol system includes a motion calculator having at least one angularcontroller which calculates a required angular displacement necessary toachieve said desired motion.
 16. The amphibious robotic device accordingto claim 15, wherein the angular control includes a yaw controllercalculating a required yaw of the device, a pitch controller calculatinga required pitch of the device and a roll controller calculating arequired roll of the device, the desired motion including at least oneof the required yaw, the required pitch and the required roll.
 17. Acontrol system for autonomously maneuvering a robotic device in at leastone of a liquid medium and a solid medium, the robotic device includinga propulsion system having a series of individually controlled legs, thecontrol system comprising: at least one visual sensor retrieving animage of an environment of the device in the medium; an image analyzingmodule receiving the image, determining a presence of an object of agiven type therein and analyzing at least one property of the object; amotion calculator determining a desired motion of the device based onthe at least one property of the object; and a controller operating thepropulsion system of the device, the controller calculating a respectiverequired leg motion of each of the legs to obtain the desired motion ofthe device and operating each of the legs based on the respectiverequired leg motion calculated, such that the robotic deviceautonomously maneuvers in said medium.
 18. The control system accordingto claim 17, wherein the desired motion of the device includes stationkeeping.
 19. The control system according to claim 17, wherein themotion calculator is programmed upon reception of the at least oneproperty to memorize a series of at least two consecutive steps, eachstep including one of movement in at least one of six degrees of freedomand station keeping, the desired motion successively corresponding toeach of the consecutive steps.
 20. The control system according to claim17, wherein the image analyzing module determines an identity of theobject from a given list of possible objects of the given type, themotion calculator determining the desired motion of the device from thelist where a different desired motion is associated with each of atleast some of the possible objects of the given type.
 21. The controlsystem according to claim 17, wherein the image analyzing moduledetermines a position of the object on the image and compares theposition to a desired position, and the motion calculator determiningthe desired motion of the device such as to change the position tocorrespond to the desired position.
 22. The control system according toclaim 21, wherein the object is moving, and the desired motion of thedevice allows the device to follow the object.
 23. The control systemaccording to claim 17, wherein the motion calculator includes at leastone angular controller which calculates a required angular displacementnecessary to achieve said desired motion.
 24. The control systemaccording to claim 23, wherein the angular control includes a yawcontroller calculating a required yaw of the device, a pitch controllercalculating a required pitch of the device and a roll controllercalculating a required roll of the device, the desired motion includingat least one of the required yaw, the required pitch and the requiredroll.
 25. The control system according to claim 24, wherein the yaw,pitch and roll controllers are PID controllers.